{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "(monitoring-models)=\n",
    "# View model monitoring results in the platform UI\n",
    "\n",
    "This tutorial illustrates the basic model monitoring capabilities of MLRun: deploying a model to a live endpoint and observing data drift. MLRun calculates the drift using the monitoring application and stores the result in the model endpoint. The UI presents the model endpoint information and stats, including the drift status column.\n",
    "\n",
    "**In this section**\n",
    "\n",
    "* [Model endpoint summary list](#model-endpoints-summary-list)\n",
    "* [Model endpoint overview](#model-endpoints-overview)\n",
    "* [Model features analysis](#model-endpoints-features-analysis)\n",
    "* [Model endpoints metrics](#model-endpoints-metrics)\n",
    "\n",
    "\n",
    "## Model endpoints summary list\n",
    "\n",
    "1. Select a project that has Model monitoring already enable.\n",
    "2. From the project dashboard, press the **Models** tile to view the models currently deployed.\n",
    "2. Press **Model Endpoints** from the menu to display a list of monitored endpoints.<br>  \n",
    "   \n",
    "The Model Endpoints summary list provides a quick view of the model monitoring data.\n",
    "\n",
    "![Model Monitoring Summary List](../_static/images/model_endpoints_main.png)\n",
    "\n",
    "The summary page contains the following fields:\n",
    "* **Name** &mdash; the name of the model endpoint\n",
    "* **Function** &mdash; the name of the related function  \n",
    "* **Version** &mdash; user configured version taken from model deployment\n",
    "* **Class** &mdash; the implementation class that is used by the endpoint\n",
    "* **Labels** &mdash; user configurable tags that are searchable\n",
    "* **First prediction** &mdash; first request for production data\n",
    "* **Last prediction** &mdash; most recent request for production data\n",
    "* **Error count** &mdash; includes prediction process errors such as operational issues (for example, a function in a failed state), as well as data processing errors\n",
    "(For example, invalid timestamps, request ids, type mismatches etc.)\n",
    "* **Drift Status** &mdash; indication of drift status (no drift (green), possible drift (yellow), drift detected (red))\n",
    "\n",
    "## Model endpoints overview\n",
    "The Model Endpoints overview pane displays general information about the selected model.\n",
    "\n",
    "![Model Endpoints Overview](../_static/images/IG_model_endpoints_overview.png)\n",
    "\n",
    "The Overview page contains the following fields:\n",
    "* **UUID** &mdash; the ID of the deployed model\n",
    "* **Model class** &mdash; the implementation class that is used by the endpoint\n",
    "* **Model artifact** &mdash; reference to the model's file location\n",
    "* **Function URI** &mdash; the MLRun function to access the model\n",
    "* **Function Tag** &mdash; the MLRun function tag\n",
    "* **Feature set** &mdash; the monitoring feature set that points to the monitoring parquet directory\n",
    "* **Last prediction** &mdash; most recent request for production data\n",
    "* **Error count** &mdash; includes prediction process errors such as operational issues (For example, a function in a failed state), as well as data processing errors\n",
    "(For example, invalid timestamps, request ids, type mismatches etc.)\n",
    "* **Accuracy** &mdash; a numeric value representing the accuracy of model predictions (N/A)\n",
    "* **Stream path** &mdash; the input and output stream of the selected model\n",
    "* **Mean TVD** &mdash; the mean value of the [Total Variance Distance](../model-monitoring/index.md#model-monitoring-applications) of the model features and labels\n",
    "* **Mean Hellinger** &mdash; the mean value of the [Hellinger Distance](../model-monitoring/index.md#model-monitoring-applications) of the model features and labels\n",
    "* **Mean KLD** &mdash; the mean value of the [KL Divergence](../model-monitoring/index.md#model-monitoring-applications) of the model features and labels\n",
    "* **Drift Actual Value** &mdash; the resulted drift value of the latest drift analysis calculation. \n",
    "* **Drift Detected Threshold** &mdash; pre-defined value to determine a drift \n",
    "* **Possible Drift Threshold** &mdash; pre-defined value to determine a possible drift\n",
    "\n",
    "```{note}\n",
    "Press **Resource monitoring** to get the relevant [Grafana Model Monitoring Details Dashboard](./monitoring-models-grafana.ipynb#model-monitoring-details-dashboard) that displays detailed, real-time performance data of the selected model. In addition, use the ellipsis to view the YAML resource file for details about the monitored resource.\n",
    "```\n",
    "\n",
    "## Model endpoints features analysis\n",
    "The Features Analysis pane provides details of the drift analysis in a table format with each feature and label in the selected model on its own line.\n",
    "\n",
    "![Model Endpoints Features Analysis](../_static/images/IG_model_endpoints_features_analysis.png)\n",
    "\n",
    "Each field has a pair of columns. The **Expected** column displays the results from the model training phase, and the **Actual** column\n",
    "displays the results from the live production data. The following fields are available:\n",
    "* **Mean**\n",
    "* **STD** (Standard deviation)\n",
    "* **Min**\n",
    "* **Max**\n",
    "* **TVD**\n",
    "* **Hellinger**\n",
    "* **KLD**\n",
    "* **Histograms**&mdash;the approximate representation of the distribution of the data. Hover over the bars in the graph for the details.\n",
    "\n",
    "## Model endpoints metrics\n",
    "\n",
    "The Metrics pane displays the pre-defined metrics for model endpoints such as `predictions_per_second` and `latency_avg_5m`. You can also define your own metrics using\n",
    "{py:meth}`mlrun.model_monitoring.applications.ModelMonitoringApplicationMetric`.\n",
    "Each metric can be seen by selecting its name from the drop-down list.\n",
    "\n",
    "![Model Monitoring Metrics](../_static/images/mm-metrics.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
